AD  A1Q789  1 


DEPARTMENT  OF  DEFENSE  AND  SERVICE  REQUIREMENTS 
FOR  HUMAN  FACTORS  R&D  IN  THE  MILITARY  SYSTEM  ACQUISITION  PROCESS 


By 

Harold  E.  Price 
Marco  Fiorello 
John  C.  Lowry 
M.  Gregory  Smith 
Jerry  S.  Kidd 

BioTechnology,  Inc. 

3027  Rosemary  Lane 
Fails  Church,  VA 


July  1980 


Air  Force  Office 
of  Scientific  Rnaerch 
Boiling  AFB,  Washington,  DC 

A 


DOD  HUMAN  FACTORS  ENGINEERING  (HFE) 
TECHNICAL  ADVISORY  GROUP  (TAG) 


DTIC 

SELECTEI 
NOV  2  5  1981 

B 


Contract  MD A  903-79-C-0553 


'distribution  statement  a 

Approved  lot  public  release* 
Distribution  Unlimited 


Via 


U.S.  ARMY  RESEARCH  INSTITUTE 
FOR  THE  BEHAVIORAL  AND  SOCIAL  SCIENCES 

5001  Eisenhower  Avenue 
Alexandria,  Virginia  22333 


Other  Supporting  Organizations 


Aerospace  Medical 
Research  Laboratory 
Qeyton,  OH 

8111  16056 


U.S.  Army  Human 
Engineering  Laboratory 
Aberdeen,  MD 


Naval  Air 

Development  Center 
Warminster,  PA 


RESEARCH  NOTE  8<b23 


DEPARTMENT  OF  DEFENSE  AND  SERVICE  REQUIREMENTS 
FOR  HUMAN  FACTORS  R&D  IN  THE  MILITARY  SYSTEM  ACQUISITION  PROCESS 


By 

Harold  E.  Price 
Marco  Fiorello 
John  C.  Lowry 
M.  Gregory  Smith 
Jerry  S.  Kidd 

BioTechnology,  Inc. 

3027  Rosemary  Lane 
Falls  Church.  VA 


DTIC 

ELECTE| 


NOV  2  5  1981 

B 


July  1980 


DISTRIBUTION  STATEMENT  A 

Approved  for  public  ieleasej 
Distribution  Unlimited 


DOD  HUMAN  FACTORS  ENGINEERING  (HFE) 
TECHNICAL  ADVISORY  GROUP  (TAG) 


Contract  MD A  903-79-C-05S3 


SECURITY  CLASSIFICATION  OF  THIS  PAGE  (When  Data  Entered) 


REPORT  DOCUMENTATION  PAGE 


RKAIJ  INSTRUCTIONS 
BEFORE  COMPLETING  FORM 


1  REPORT  NUMBER 

Research  Note  80-23 


4  TITLE  (and  Subtitle) 

DEPARTMENT  OF  DEFENSE  AND  SERVICE  REQUIREMENTS 
FOR  HUMAN  FACTORS  IN  THE  MILITARY  SYSTEM 
ACQUISITION  PROCESS 


7.  AUTHORS 

Harold  E.  Price,  Marco  Fiorello,  John  C.  Lowry, 
M.  Gregory  Smith,  and  Jerry  S.  Kidd 


2.  GOVT  ACCESSION  NO  3  RECIPIENT'S  CATALOG  NUMBER 


6  TYPE  OF  REPORT  A  PERIOD  COVERED 

Final 

September  1979  -  July  1980 


6  PERFORMING  ORG  REPORT  NUMBER 


8  CONTRACT  OR  GRANT  NUMBERO; 

MDA  903-79-C-0553 


9  PERFORMING  ORGANIZATION  NAME  AND  ADDRESS 

BioTechnology ,  Inc. 

3027  Rosemary  Lane 
Falls  Church,  VA  22042 


11  CONTROLLING  OFFICE  NAME  AND  ADDRESS 

U.S.  Army  Research  Institute  for  the 

Behavioral  and  Social  Sciences 

5001  Eisenhower  Ave.,  Alexandria,  VA  22333 


14  MONITORING  AGENCY  NAME  &  ADDRESS 
(if  different  from  Controlling  Office) 


10  PROGRAM  ELEMENT.  PROJECT.  TASK 
AREA  &  WORK  UNIT  NUMBERS 


2Q162722A765 


12.  REPORT  DATE 

July  1980 


13.  NUMBER  OF  PAGES 

63 


15  SECURITY  CLASS,  (of  this  report) 

Unclassified 


15*.  DECLASSIFICATION/DOWNGRADING 
SCHEDULE 


16  DISTRIBUTION  STATEMENT  (of  thu  Report) 


Approved  for  public  release;  distribution  unlimited 


17  DISTRIBUTION  STATEMENT  (of  the  abstract  entered  in  Block  20,  if  different  from  Report) 


18  SUPPLEMENTARY  NOTES 

This  research  note  is  a  companion  document  to  ARI  Technical  Report  No.  476, 
entitled;  "The  Contribution  of  Human  Factors  in  Military  System  Development; 
Methodological  Considerations." 


19  KEY  WORDS  (Continue  on  reverse  tide  if  necessary  and  identify  by  block  number) 

Human  Factors  Human  Engineering 

Military  System  Development  Service  Regulations 

Department  of  Defense  Directives  Service  Instructions 


20  ABSTRACT  (Continue  on  reverse  tide  if  necessary  and  identify  by  block  number ) 

^  This  research  note  presents  a  thorough  analysis  of  Department  of  Defense 
and  3ervice-level  policy  requirements  for  human  factors  in  the  military  system 
acquisition  process.  It  is  intended  as  a  companion  document  to  ARI  Technical 
Report  No.  476,  and  as  such  provides  detailed  indoctrination  into  both  the 
system  development  process  and  human  factors  R6D.  It  will  be  particularly 
useful  to  those  who  require  a  greater  understanding  of  this  process. 


(continued) 


FORM  *473  EDITION  OF  1  NOV  65  IS  OBSOLETE 

1  JAN  73  '  *  S/NO102-014-BB0I  UNCLASSIFIED 


SECURITY  CLASSIFICATION  OF  THIS  PAGE  (When  Date  Entered) 


•K, 


SECURITY  CLASSIFICATION  OF  THIS  FAGE 


la  Entered) 


Beginning  with  an  explanation  of  each  system  development  phase, 
hisnan  factors  R&D  requirements  are  then  integrated  into  the  system 
development  framework.  This  is  accomplished  through  direct  referral  to 
Department  of  Defense  Directives,  Specifications,  and  Standards. 

Recognizing  that  differences  occur  in  Army,  Navy,  and  Air  Force  implementa¬ 
tion  of  human  factors  R&D,  formal  service  documentation  is  presented  for 
human  factors  in  each  phase  of  system  development.  These  documents 
include  service  regulations  and  instructions .  Formal  service  documentation 
provides  only  that  human  factors  requirements  be  implemented,  while  not 
setting  forth  a  particular  implementation  plan.  Informal  service 
documentation  is  presented  to  illustrate  the  processes  by  which  human 
factors  is  implemented  in  system  development  programs  initiated  by  each 
service. 


SECURITY  CLASSIFICATION  OF  THIS  FAGE  ffhrn  Data  Entered) 


CONTENTS 


Page 

Department  of  Defense  and  Service  Requirements 
for  Human  Factors  R&D  in  the  Military  System 

Acquisition  Process  .  1 

Overview  of  the  System  Acquisition  Process  .  1 

Mission  Analysis  Phase  .  1 

Concept  Development  Phase  .  5 

Demonstration  and  Validation  Phase  .  12 

Full-Scale  Development  Phase  .  15 

Production  and  Deployment  Phase  .  22 

Deployment  Preparation  .  22 

Retrofit/Improvements  .  22 

Human  Factors  R&D:  DOD  Requirements . 24 

Mission  Analysis  Phase  .  24 

Concept  Development  Phase  .  27 

Decision  Coordinating  Paper  .  31 

Demonstration  and  Validation  Phase  .  33 

MIL-STD-1472B  .  35 

Full-Scale  Development  Phase  .  38 

Production  and  Deployment  Phase  .  41 

Human  Factors  R&D:  Service  Requirements  .  43 

Overall  Military  System  Acquisition 

Human  Factors  R&D  Requirements . 44 

Service  Requirements  for  Human  Factors  R&D 

in  Mission  Analysis  .  46 

Service  Requirements  for  Human  Factors 

in  Concept  Development  .  47 

Service  Requirements  for  Human  Factors  R&D 

in  the  Demonstration  and  Validation  Phase  .  48 

Service  Requirements  for  Human  Factors  R&D 

in  the  Full-Scale  Development  Phase  .  50 

Service  Requirements  for  Human  Factors  R&D 

Through  Production  and  Post- Production  Phases  ....  51 

Informal  Documents  .  52 

Conclusions . 55 

References . 57 


EXHIBITS 


u 


> 


» 


» 


» 


> 


Exhibit  Page 

1  The  Phases  cf  the  Military's  Major 

System  Acquisition  Model  .  2 

2  Events  During  Mission  Analysis  Phase  .  3 

3  Events  During  Concept  Development  Phase  .  6 

4  Events  During  the  Demonstration  and 

Validation  Phase  .  13 

5  Events  During  the  Full-Scale 

Development  Phase  .  16 

6  Events  During  the  Production 

and  Deployment  Phase  .  23 


DEPARTMENT  OF  DEFENSE  AND  SERVICE  REQUIREMENTS  FOR 
HUMAN  FACTORS  R&D  IN  THE  MILITARY  SYSTEM  ACQUISITION  PROCESS 


The  discussion  contained  herein  provides  a  detailed  expansion 
of  DOD  and  service  requirements  for  the  system  acquisition  process 
and  for  human  factors  R&D  associated  with  this  process.  This 
discussion  is  meant  to  support  certain  sections  of  an  ARI  technical 
report  entitled  "The  Contribution  of  Human  Factors  in  Military 
System  Development"  (Price  et  al. ,  1980) .  Specifically,  it 
supports  the  summary  and  review  of  those  topics  to  be  found  in 
the  first  section  of  Chapter  3  of  that  document.  This  material 
is  recommended  especially  for  those  readers  who  lack  confidence 
in  their  understanding  of  the  system  development  process  and  of 
the  specific  efforts  involved  in  each  phase  (especially  as  regards 
human  factors  R&D)  . 

Overview  of  the  System  Acquisition  Process 

As  stated  in  Chapter  3,  DOD  directives  5000.1,  5000.2,  and 
5000.3  provide  military  implementation  procedures  for  the  guide¬ 
lines  and  policies  outlined  in  the  Office  of  Management  and 
Budget  Circular  A-109.  The  crux  of  these  guidelines  and  policies 
is  contained  in  the  arrangement  of  phases  within  the  overall 
system  development  cycle.  Exhibit  1  depicts  an  overall  military 
major  system  acquisition  model  in  order  to  provide  the  reader 
with  an  overview  of  the  developmental  process  as  it  exists  today. 
The  following  discussion  of  the  model  will  be  on  a  phase-by-phase 
basis,  to  be  followed  by  an  integration  of  the  basis  for  human 
factors  R&D  (HF  R&D)  at  both  the  DOD  and  service  level,  with  the 
system  development  process  as  outlined. 

Mission  Analysis  Phase 

1.  Military  and  Political  Changes,  R&D.  Events  during 
the  Mission  Analysis  Phase  are  shown  in  Exhibit  2.  In  this 
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Events  During  Mission  Analysis  Phase 
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phase  the  DOD  Component  Head  (Secretary  of  the  Army,  Navy,  or 
Air  Force)  is  responsible  for  maintaining  basic  research  and 
development  programs  to  help  support  future  system  needs. 

However,  the  identification  of  technological  breakthroughs  can 
come  from  research  by  government  sponsored  laboratories,  the 
private  sector,  or  universities. 

The  other  major  activity  in  this  phase  is  threat  analysis. 

The  analyses  are  conducted  on  world  military  and  political 
changes.  When  a  serious  threat  or  a  technological  breakthrough 
has  been  identified  by  the  Secretary  of  Defense  (SECDEF)  or  a 
DOD  Component  Head,  the  DOD  Component  Head  submits  a  Mission 
Element  Need  Statement  (MENS)  to  the  Secretary  of  Defense. 

2.  MENS.  The  MENS  should  describe  the  mission,  how  the 
mission  is  to  be  supported,  and  provide  justification  for 
initiation  of  a  new  major  system  acquisition.  Specifically, 

Directive  5000.2  (paragraph  C.l)  states  that  the  MENS  shall: 

Identify  the  mission  area  and  state  the  need  in 
terms  of  the  mission  element  task  to  be  performed. 

The  mission  need  shall  not  be  stated  in  terms  of 
capabilities  and  characteristics  of  a  hardware  or 
software  system. 

Assess  the  projected  threat  through  the  time  frame 
the  capability  is  required. 

Identify  the  existing  DOD  capability  to  accomplish 
the  mission. 

Assess  the  need  in  terms  of  a  deficiency  in  the 
existing  capability,  a  projected  physical  obso¬ 
lescence,  or  a  technological  or  cost  savings 
opportunity. 
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•  State  the  known  constraints  to  apply  to  any  acceptable 
solution  including  operational  and  logistics  consider¬ 
ations,  requirements  for  NATO  standardization  or 
interoperability,  limits  on  the  resource  investment 

to  be  made,  timing,  etc.  These  constraints  will 
constitute  boundary  conditions  for  the  exploration 
of  alternative  solutions. 

•  Assess  the  impact  of  not  acquiring  or  maintaining  the 
capability . 

•  Provide  a  program  plan  to  identify  and  explore 
competitive  alternative  systems  extending  through 

to  the  next  Milestone  decision.  Include  the  planning 
to  establish  a  system  program  office. 

After  the  MENS  has  been  drawn  up,  it  is  submitted  to  the 
Secretary  of  Defense  for  approval  for  concept  development. 

3.  Milestone  0.  If  the  Secretary  of  Defense  gives  approval 
at  Milestone  0,  concept  development  is  begun  and  alternative 
solutions  to  the  mission  need  are  explored. 

Concept  Development  Phase 

The  events  in  the  Concept  Development  Phase  are  shown  in 
Exhibit  3.  A  discussion  of  these  events  follows. 

1.  Program  Manager.  If  the  program  is  approved,  the  DOD 
Component  Head  assigns  a  Program  Manager  to  achieve  the  program 
objectives.  No  change  of  Program  Managers  should  occur  prior 
to  Milestone  1. 

Directive  5000.2  states  that  the  Program  Manager  shall: 
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Events  During  Concept  Development  Phase 


•  Develop  an  acquisition  strategy  for  the  total  program. 


•  Make  maximum  use  of  competition  for  achieving  program 
objectives. 

•  Make  tradeoff  decisions  in  system  capability,  cost, 
schedule,  and  risk  within  stated  ranges  as  limited  by 
threshold  values.  Program  management  includes  the 
responsibility  to  determine  whether  a  program  should 
be  continued  or  terminated  and  to  recommend  the 
appropriate  action. 

•  Make  every  effort  to  prevent  the  expenditure  of 
resources  to  achieve  unnecessary  performance  and 
schedule  requirements.  Meaningful  relationships 
shall  be  established  among  need,  urgency,  risk,  and 
worth  to  allow  practical  tradeoffs  among  system 
capability,  cost,  and  schedule.  The  Program  Manager 
shall  take  positive  action  to  continually  assess 
program  risk  areas  and  to  make  or  propose  tradeoffs 
in  performance,  cost,  and  schedule  to  achieve  the 
best  balance. 

The  Program  Manager  assembles  a  team  to  consider  alternative 
ways  to  meet  the  mission  need.  The  team  carries  out  studies  on 
the  alternatives  to  determine  their  feasibility,  effectiveness, 
cost  of  ownership,  etc. 

The  hardware  in  this  phase  is  in  breadboard  configurations. 
These  experimental  prototypes  are  built  to  determine  whether  the 
engineering  concept (s)  is  feasible  or  not. 

2,  Developmental  Test  and  Evaluation  (DT) .  DT  is  conducted 
to  verify  that  technical  performance  specifications  and  objectives 
are  being  met.  Where  appropriate,  DT  includes  testing  of 
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components,  subsystems  hardware/software  integration,  prototypes, 
complete  systems,  and  the  compatibility  with  existing  or  future 
equipment  or  systems.  In  this  phase,  DT  occurs  where  appropriate 
to  aid  in  selecting  alternative  system  concepts. 

3.  Operational  Test  and  Evaluation  (OT) .  OT  is  conducted 
to  determine  a  system's  effectiveness  and  suitability.  OT  is  not 
generally  conducted  in  this  phase,  but  can  occur  to  assess  the 
operational  impact  of  the  proposed  concepts  and  to  assist  in 
selecting  the  preferred  alternatives. 

4.  Decision  About  Alternatives.  After  studying  the 
alternatives,  the  DOD  Component  Head  makes  a  recommendation. 

That  recommendation  (based  on  Directive  5000.2)  could  be  that 
the  next  phase,  Demonstration  and  Validation,  should: 

•  involve  several  alternatives;  or 

•  be  limited  to  a  single  concept;  or 

•  involve  alternative  subsystems  only  and  not  be 
conducted  at  the  system  level;  or 

•  there  should  be  no  competitive  systems  demonstration, 
and  the  program  should  proceed  directly  into  full- 
scale  development. 

5.  Decision  Coordinating  Paper  (DCP) .  After  making  a 
decision,  a  DCP  is  prepared  recommending  the  preferred  alternatives 
for  the  next  developmental  phase.  The  principal  purpose  of  the 
DCP  is  to  support  the  Defense  System  Acquisition  Review  Council 
(DSARC)  and  the  Service  Acquisition  Review  Council  ((S)SARC) 
review  and  to  aid  the  Secretary  of  Defense  at  the  decision-making 
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Milestones.  Each  service  has  its  own  (S)SARC.  The  Army's  council 
is  ASARC;  the  Navy's  NSARC;  and  the  Air  Force's  AFSARC. 

Directive  5000.2  (Enclosure  2,  II)  states  that  the  DCP  will 
include : 

•  Update  of  the  MENS . 

•  Description  of  the  alternative  programs,  including 
anticipated  performance  information. 

•  A  summary  of  the  acquisition  strategy. 

•  Short-  and  long-term  business  planning  information. 

•  Program  structure  and  management  plan  to  include 
security  classification  guidance. 

•  Areas  of  program  uncertainty  (excluding  technical  risks) 
and  the  probable  impact. 

•  Each  DCP  prepared  for  Milestones  1  and  2  shall  contain 
a  Technology  Assessment  Annex  (TAA)  that  will  identify 
any  area  of  technological  risk  remaining  in  the  program 
and  describe  plans  for  addressing  these  risks.  The  TAA 
shall  be  prepared  by  the  Program  Manager,  assisted  by  a 
laboratory  or  laboratories  selected  for  this  purpose. 

•  A  resource  annex  for  each  program  alternative.  The 
annex  shall  include  cost,  production,  and  inventory/ 
objective  data. 

•  A  one-page  logistics  annex  for  Milestones  1,  2,  and  3. 
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•  DCPs  prepared  for  Milestones  2  and  3  shall  contain  firm 
program  schedule,  cost,  and  performance  information. 
Program  thresholds  shall  be  established  for  selected 
performance,  cost,  and  schedule  factors  representing 
acceptable,  projected  variances  at  program  completion 
and  fiscal  year  thresholds  for  the  same  cost  and  schedule 
factors  to  represent  acceptable  variances  at  the  end  of 
each  fiscal  year. 

•  Test  and  evaluation  planning  and  status. 

•  Program  issues,  including  their  assessment. 

•  DSARC  and  (S)SARC  results  and  recommendations. 

•  Secretary  of  Defense  decisions  and  direction. 

6.  DSARC  Membership.  Directive  5000.2  (Enclosures  I,  II, 
and  III)  states  who  shall  be  the  members,  participants,  and 
advisors  of  the  DSARC.  The  DSARC  members  shall  be: 


•  Defense  Acquisition  Executive  (Chairman) 

•  Director  of  Defense  Research  and  Engineering 

•  Assistant  Secretary  of  Defense  (Installations  and  Logistics) 

•  Assistant  Secretary  of  Defense  (Comptroller) 

•  Assistant  Secretary  of  Defense  (Intelligence) 

•  Director  of  Planning  and  Evaluation 

•  Director  of  Telecommunications  and  Command  and  Control 
Systems 

•  Other  OSD  staff  principals  when  essential  to  the  program 
under  review. 
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The  DSARC  participants  and  advisors  shall  be: 

•  Joint  Chiefs  of  Staff  representative 

•  Deputy  DDR&E  (T&E) 

•  Chairman  of  the  Cost  Analysis  Improvement  Group 

•  DOD  Component  Head. 

7.  Milestone  1  Issues.  DSARC  I/(S)SARC  I  must  address  the 
following  issues  (Directive  5000.2,  Enclosure  2,  IV. A)  for 
Milestone  1: 

•  The  mission  element  task  to  be  accomplished  is  reaffirmed 
to  be  essential. 

•  An  updated  threat  assessment. 

•  The  alternative  system  design  concepts  adequately  reflect 
the  technology  base  and  provide  an  acceptable  competitive 
environment. 

•  Foreign  developments  have  been  considered. 

•  The  alternatives  recommended  for  demonstration  and 
validation  meet  the  mission  element  needs. 

•  The  established  program  constraints  remain  valid. 

•  The  projected  resource  investment  for  the  selected 
alternatives  and  other  characteristics  related  to  the 
alternatives  are  consistent  with  the  stated  constraints. 

•  Operational  and  logistical  considerations  are  adequate. 

•  Use  of  available  subsystems  and  existing  military  and 
commercial  hardware  and  software  is  adequately  considered. 

•  The  acquisition  strategy  is  complete,  effectively  inte¬ 
grates  the  program  technical,  business,  and  management 
elements,  and  supports  the  achievement  of  program  goals 
and  objectives. 
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•  Short-  and  long-term  business  planning  effectively 
supports  the  acquisition  strategy. 

•  Producibility  and  areas  of  production  risks  have  been 
adequately  considered. 

•  Joint-Services,  interoperability,  and  multi-national 
considerations  are  adequately  treated  in  the  planning. 

•  NATO  standardization  and  interoperability  requirements 
have  been  adequately  considered. 

•  Risk  and  uncertainty  areas  are  identified  and  adequately 
treated  in  the  planning. 

•  Environmental  considerations  are  adequate. 

•  Planning  and  schedules  for  preparation  of  the  Test  and 
Evaluation  Master  Plan  (TEMP)  are  adequate. 

•  The  program  management  structure. 

8.  Milestone  1.  Following  the  DSARC  I/(S)SARC  I  reviews, 
the  Secretary  of  Defense  is  presented  one  or  more  alternatives 
for  system  demonstration.  These  recommendations  are  contained 
in  the  DCP.  Following  the  Secretary  of  Defense  review,  the 
Secretary  of  Defense  either  concurs  in  the  need  for  demonstration 
and  validation,  sends  the  program  back  for  further  consideration, 
or  cancels  the  program. 

Demonstration  and  Validation  Phase 

The  events  in  the  Demonstration  and  Validation  Phase  are 
shown  in  Exhibit  4.  Demonstration  and  validation  provide  a 
basis  for  selection  of  a  system  for  full-scale  development. 

The  demonstrations  should  be  with  full-scale  prototypes  in 
realistic  operating  environments.  If  system  demonstrations  are 
not  practical  or  feasible,  critical  subsystems  will  undergo 
competitive  demonstrations. 
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Because  the  developing  systems  will  not  be  adequately 
defined  before  the  completion  of  this  phase,  performance,  cost, 
and  schedule  estimates  are  not  considered  firm  prior  to  the 
Milestone  2  decision. 

1.  Test  and  Evaluation  Master  Plan  (TEMP).  An  initial 
version  of  the  overall  test  and  evaluation  plan  should  be  prepared 
as  early  as  possible.  An  Office  of  Secretary  of  Defense-approved 
TEMP  is  a  prerequisite  for  Milestone  2.  TEMP  should  identify  and 
integrate  objectives,  responsibilities,  resources,  and  schedules 
for  all  tests  and  evaluations. 

2.  Developmental  Test  and  Evaluation  I.  DT  I  is  conducted 
to  aid  in  the  selection  of  the  preferred  technical  approach,  to 
show  that  technical  risks  have  been  identified,  and  to  demonstrate 
that  solutions  are  available. 

3.  Operational  Test  and  Evaluation  I.  The  first  OT  is 
generally  conducted  in  this  phase;  it  is  to  be  conducted  in  as 
realistic  an  environment  as  possible,  so  as  to  provide  information 
about  the  operational  effectiveness  and  suitability  of  each  of  the 
candidate  systems.  OT  includes  gaining  information  on  a  system's 
survivability /vulnerability ,  transportability,  reliability, 
maintainability,  safety,  human  factors,  logistic  supportability , 
and  training  requirements  as  well  as  information  on  personnel 
requirements,  doctrine,  and  tactics. 

In  order  not  to  expend  unnecessary  resources,  any  duplicate 
tests  scheduled  for  DT  and  OT  should  be  combined  into  a  single 
test.  The  test  must  be  coordinated,  planned,  and  executed  to 
provide  all  needed  information  to  the  involved  agencies. 
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In  each  service,  there  is  an  agency  charged  with  the  conduct 
of  OT.  This  agency  is  separate  and  distinct  from  the  developing, 
procuring,  or  using  command.  This  separate  agency  is  to: 

•  participate  in  the  planning  of  those  portions  of  DT  that 
pertain  to  the  accomplishment  of  OT  objectives; 

•  review  the  results  of  DT  that  pertain  to  the  accomplishment 
of  OT  objectives; 

•  insure  that  all  OTs  are  effectively  planned  and  conducted; 
and 

•  provide  to  (S)SARC  the  results  of  OTs  just  completed,  and 
the  adequacy  of  planned  OTs. 

4.  Update  DCP.  After  demonstration  and  validation  is 
completed,  the  DOD  Component  Head  updates  the  DCP  to  recommend 
one  system  for  full-scale  development  and  production. 

5.  Milestone  2.  If  the  program  is  to  continue,  the  Secretary 
of  Defense  approves  the  selection  of  a  system  for  full-scale 
development  and  the  limited  production  of  enough  items  for 
operational  test  and  evaluation. 

Full-Scale  Development  Phase 

The  events  for  the  Full-Scale  Development  Phase  are  shown  in 
Exhibit  5.  There  is  to  be  no  change  of  Program  Managers  during 
full-scale  development  or  prior  to  a  Milestone  3  decision.  A 
firm,  detailed  logistics  plan  is  to  be  established  early  in  this 
phase.  Subsystems  are  not  to  be  fully  developed  until  a  system 
has  been  selected  for  full-scale  development. 
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Exhibit  5 


Events  During  the  Full-Scale  Development  Phase 
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1.  Proposals  and  Prototypes.  Contractors  are  required  to 
submit  firm  proposals  for  their  systems  for  full-scale  development 
and  for  initial  production  items.  DOD  will  supply  contractors 
with  the  factors,  criteria,  and  conditions  to  be  used  in  the 
evaluation  and  selection  of  the  system.  Contractors  are  to  tailor 
the  standards  and  specifications  to  the  particular  system  under 
development.  Firm  acquisition  and  ownership  costs  are  to  be 
established.  After  being  selected,  the  contractor  proceeds  with 
the  building  of  the  prototype (s)  for  DT/OT  testing. 

The  DCP  for  Milestone  1  should  contain  a  Technology  Assess¬ 
ment  Annex  that  identifies  any  area  of  technological  risk  that 
remains  in  the  program  and  ways  to  address  that  risk. 

DSARC  II/ ( S ) SARC  II  reviews  are  to  address  the  following 
issues  for  the  Milestone  2  decisions  (Directive  5000.2, 

Enclosure  2,  IV,  B) : 

•  The  mission  element  task  to  be  accomplished  is  reaffirmed 
and  the  threat  updated. 

•  The  system  selected  meets  the  mission  element  needs,  is 
cost-effective,  and  is  acceptable  within  stated  constraints. 

•  NATO  standardization  and  interoperability  requirements 
are  satisfied. 

•  The  demonstration  and  validation  results  support  the 
system  recommended. 

•  System  tradeoffs  have  produced  the  most  effective  balance 
in  cost,  performance,  and  schedule,  including  operational 
and  logistical  considerations. 
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•  Uncertainties  and  risks  have  been  identified  and  are 
acceptable;  planning  to  resolve  the  remaining 
uncertainties  and  risks  is  adequate.  Realistic  fall¬ 
back  actions  and  alternatives  have  been  established. 

•  The  acquisition  strategy  has  been  updated,  effectively 
supports  achievement  of  program  objectives,  and  is 
being  executed  in  the  conduct  of  program  management. 

•  Short-  and  long-term  business  planning  supports  the 
strategy.  Contract  types  are  consistent  with  the 
program  characteristics,  risks,  uncertainty,  and 
strategy. 

•  Design- to- cost  and  life  cycle  cost  requirements  are 
realistic  and  effective  in  achieving  cost  objectives. 

•  Cost,  performance,  and  schedule  estimates  and  related 
thresholds  have  been  thoroughly  reviewed,  are  well 
defined,  and  are  consistent  with  risks  involved.  These 
values  shall  be  established  as  firm  estimates. 

•  Action  to  submit  the  initial  Selected  Acquisition 
Report  (SAR)  is  complete. 

•  Planning  for  selection  of  major  subsystems  is  clearly 
stated,  provides  for  sustained  competition  to  the 
maximum  extent  feasible,  and  accepts  the  use  of  existing 
military  and  commercial  hardware  and  software  where 
appropriate.  Foreign  developments  have  been  considered. 

•  Demonstration  and  validation  testing  and  evaluations 
have  been  completed  and  results  support  the 
recommendations . 
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•  Electronic/infrared/optical  counter-countermeasure 
performance  requirements  have  been  identified. 


•  Producibility  considerations  and  areas  of  production 
risks  have  been  reviewed  and  the  results  found 
acceptable. 

•  Requirements  have  been  established  for  long- lead 
procurement  items  and  initial  limited  production  to 
support  operational  test  and  evaluation  needs  for 
the  verification  of  production  engineering  and  design 
maturity,  and  to  establish  the  production  base. 

•  The  Test  and  Evaluation  Master  Plan  (TEMP)  identifies 
and  integrates  the  testing  and  evaluation  to  be 
accomplished  prior  to  the  Milestone  2  and  3  program 
decision  points. 

•  Requisites  for  the  Milestone  3  production  and  deployment 
decision  including  operational  and  logistical  support 
have  been  established. 

•  The  program  management  structure  and  plan  are  sound  and 
adequately  supported. 

2.  Developmental  Test  and  Evaluation  II.  Adequate  develop¬ 
mental  testing  and  evaluation  shall  be  done  to  insure  that: 

•  the  engineering  is  reasonably  complete; 

•  all  significant  design  problems  (e.g.  ,  vulnerability, 
availability,  human  factors,  logistic  supportability ) 
have  been  identified,  and  that 

•  there  are  solutions  to  the  problems. 
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3.  Operational  Test  and  Evaluation  II.  This  phase  of 
operational  test  and  evaluation  should  provide  a  valid  estimate 

of  operational  effectiveness  and  suitability.  OT  II  would  include 
tests  on  operating  instructions,  documentation,  publication,  and 
handbooks . 

4.  Update  DCP.  After  the  tests  and  evaluation,  the  DCP 
is  again  updated  to  recommend  the  system  for  production  and 
deployment. 

5.  PS ARC  III/ ( S ) SARC  II.  Directive  5000.2  (Enclosure  2, 

IV.  C)  states  that  the  DSARC  III/(S)SARC  III  reviews  are  to 
address  the  following  issues  for  the  Milestone  3  decision: 

•  The  mission  element  task  to  be  accomplished  is 
reaffirmed  and  the  threat  updated. 

•  The  development  has  progressed  satisfactorily,  and 
the  initial  operational  test  and  evaluation  results 
support  a  decision  to  proceed  with  production  and 
deployment. 

•  The  acquisition  strategy  has  been  updated  and  is 
being  executed. 

•  Business  planning  supports  the  acquisition  strategy 
and  provides  flexibility  for  production  rates  and 
quantities  when  options  are  used. 

•  Schedule  and  cost  estimates  are  realistic  and 
acceptable  including  support  and  operating  costs. 
(Reference  DOD  Directive  5000.4) 

•  Design- to-cost  and  life  cycle  .cost  requirements  are 
realistic  and  effective  in  achieving  cost  objectives. 
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•  The  system  is  cost-effective  and  affordable  and  remains 
the  best  alternative. 

•  Tradeoffs  have  been  made  to  balance  cost,  schedule,  and 
performance  effectively. 

•  Program  and  fiscal  year  thresholds  are  reaffirmed. 

•  Production  quantity  requirements  are  valid. 

•  Issues  concerning  production,  producibility ,  quality 
assurance,  and  facilities  are  identified  and  managed 
satisfactorily. 

The  program  management  structure  and  plan  are  sound 
and  adequately  supported. 

•  Major  problems  are  identified  and  satisfactorily  resolved. 

•  NATO  standardization  and  interoperability  requirements 
have  been  satisfied. 

•  Requisites  for  future  production  decisions  have  been 
defined,  and  competition  has  been  considered  through 
second  source,  etc. 

•  Planning  for  deployment  is  adequate,  including  manpower 
and  training  logistics  readiness  and  operational  con¬ 
siderations  including  integration  with  existing 
operational  systems. 

•  Assessment  of  support  subsystems  to  meet  needs  of  initial 
operational  units  and  planning  to  meet  any  deficiencies 
are  carried  out. 

•  Production  readiness  review  is  completed;  contractor  has 
adequate  capability  to  manufacture  the  system. 
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6.  Milestone  3.  At  Milestone  3,  the  Secretary  of  Defense 
can  cancel,  approve,  or  have  further  studies  conducted  on  the 
system. 

Production  and  Deployment  Phase 

The  events  in  the  Production  and  Deployment  Phase  are  shown 
in  Exhibit  6.  If  the  system  is  approved  for  the  Production  and 
Deployment  Phase,  then  a  decision  has  to  be  made  for  either 
limited  or  full  production.  Generally,  if  more  testing  is 
required,  the  decision  will  be  for  limited  production;  if  there 
are  no  or  few  outstanding  issues  to  be  resolved,  the  decision 
will  be  for  full  production. 

1.  Developmental  Test  and  Evaluation  III.  If  DT  III  is 
required,  adequate  testing  shall  be  done  to  resolve  any  out¬ 
standing  issues  or  for: 

•  product  improvements; 

•  operational  modification  to  meet  identified  threat  change; 
or 

•  changes  to  reduce  system  life  cycle  costs. 

2.  Operational  Test  and  Evaluation  III.  OT  will  continue 
as  necessary  to  refine  operational  effectiveness  estimates,  to 
evaluate  changes,  and  to  re-evaluate  the  system  in  new  environ¬ 
ments  or  against  new  threats. 

Deployment  Preparation 

Production  phasing,  production  management,  production 
management  transfer,  and  full-scale  logistics  support  implementa¬ 
tion  are  made  firm  for  deployment. 

Retrofit/Improvements 

After  the  items  are  fielded,  suggestions  and/or  complaints 
can  result  in  product  changes. 


Human  Factors  R&D:  DOD  Requirements 
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Evidence  of  the  growing  awareness  of  the  need  of  human 
factors  R&D  in  the  system  acquisition  cycle  can  be  seen  in  the 
promulgation  of  governmental  and  service- level  human  factors 
requirements  and  regulations.  The  purpose  of  this  section  is 
to  review  and  integrate  the  human  factors  requirements  from 
governmental- level  documents  that  pertain  to  all  phases  of  the 
system  acquisition  cycle. 

Directives  5000.1,  5000.2,  and  5000.3  give  the  requirements 
for  any  major  governmental  acquisition.  In  the  sections  below, 
relevant  human  factors  requirements  have  been  excerpted  from  the 
directives  and  discussed  in  the  chronological  phases  of  system 
development.  In  addition,  excerpts  from  MIL-STD-1472B  and  MIL-H- 
46855B  are  used  to  complement  the  human  factors  R&D  governmental 
requirements  for  system  development. 

Mission  Analysis  Phase 

Each  DOD  Component  Head  is  responsible  for  maintaining 
on-going  analyses  to  identify  new  threats  or  technological 
breakthroughs.  When  either  are  identified,  the  DOD  Component 
Head  prepares  a  Mission  Element  Needs  Statement  (MENS)  that 
describes  the  mission,  how  the  mission  is  to  be  supported,  and 
provides  justification  of  a  new  major  system  acquisition.  In 
addition  to  providing  the  justification,  the  MENS  is  to  provide 
the  planning  for  a  system  program  office  and  to  detail  a  program 
plan  to  explore  alternative  systems  to  meet  the  threat  or  to 
maximize  the  benefit  of  a  technological  breakthrough. 

The  MENS  must  address  several  important  human  factors  R&D 
issues.  Directive  5000.2  requires  that  the  MENS: 


•  State  the  known  constraints  to  apply  to  any  acceptable 
solution  including  operational  and  logistics  con¬ 
siderations,  requirements  for  NATO  standardization 

or  interoperability,  limits  on  the  resource  investment 
to  be  made,  timing,  etc.  These  constraints  will 
constitute  the  boundary  conditions  for  the  exploration 
of  alternative  solutions.  (5000.2,  IV.C.l.e) 

Man  is  going  to  play  a  role  in  the  new  projected  mission  either 
in  carrying  out  the  mission  or  supporting  the  mission  or  both. 
HF  R&D  will  be  needed  to  determine  the  most  effective  role  of 
man  in  the  system  and  to  determine  if  the  proposed  mission 
requires  unrealistic  human  performance  standards  or  unrealistic 
operational  and  logistical  requirements.  Determining  realistic 
human  performance  constraints  will  impose  additional  boundary 
conditions  and  eliminate  consideration  of  and  the  building  of 
systems  in  which  man  cannot  properly  perform  to  the  extent 
intended.  With  proper  HF  R&D,  the  correction  of  such  design 
errors  can  occur  at  the  program  beginning  and  not  downstream 
where  changes  are  more  difficult  to  implement  and  more  costly 
to  make.  In  the  worst  case,  the  design  errors  are  not  detected 
until  after  the  system  is  fielded.  In  such  a  case,  a  seemingly 
superior  system  always  will  operate  in  a  degraded  mode  as  long 
as  the  system  does  not  account  for  man's  capabilities  and 
limitations.  If  man's  capabilities  are  not  maximized,  the 
system  will  be  suboptimal.  If  man's  limitations  are  not  taken 
into  account,  man  will  be  the  bottleneck.  The  removal  of  the 
bottleneck  generally  means  costly  system  redesign. 

Another  responsibility  of  the  DOD  Component  Head  is  to 
maintain  a  research  program  to  achieve  technological  break¬ 
throughs  and  develop  data  bases  for  use  in  future  systems. 

•  The  ability  to  achieve  required  system  capabilities 
within  acceptable  risk  and  cost  is  dependent  upon  a 
strong  and  usable  technology  base.  The  Head  of  each 


DOD  component  is  responsible  for  assuring  continued 
technology  advancement  both  in  product  and  in  manufac¬ 
turing  technology  to  support  future  system  developments. 
This  technology  base  shall  be  maintained  by  the  DOD 
Components  .  .  .  .  (5000.1,  IV. G) 


The  directives  clearly  state  the  need  for  early  test  and 
evaluation,  for  early  determination  of  probable  operational 
system  effectiveness,  and  for  determining  what  resource 
constraints  may  exist.  Of  interest  to  HF  R&D  are  tests  and 
evaluation  of  the  effectiveness  of  the  alternative  roles  of  man, 
analyzing  similar  systems  for  their  operational  effectiveness, 
and  developing  systems  that  maximize  man's  capabilities  and 
minimize  his  limitations. 

•  Test  and  evaluation  shall  be  commenced  as  early  as 
possible  and  conducted  throughout  the  system  acquisition 
process  as  necessary  to  assess  and  reduce  the  acquisition 
risks  and  to  evaluate  the  operational  effectiveness  and 
operational  suitability  of  the  system  being  developed. 
Meaningful  critical  issues,  test  criteria,  and  measures 
of  effectiveness  related  to  the  satisfaction  of  mission 
need  shall  be  established  prior  to  the  commencement  of 
tests.  (5000.3,  C.l.a) 

•  Before  the  initiation  of  development  of  a  new  system, 
test  and  evaluation  using  existing  systems,  or  modifi¬ 
cations  thereto,  should  be  conducted  when  appropriate, 
to  help  define  the  military  need  and  operational  concept 
for  the  proposed  new  system  and  to  estimate  its 
operational  effectiveness  and  operational  suitability. 
(5000.3,  C.l.d) 

•  Dependence  on  subjective  judgment  concerning  system 
performance  shall  be  kept  to  a  minimum  during  testing. 

To  the  extent  permitted  by  resource  constraints  and  the 
need  for  realistic  test  environments,  appropriate  test 
instrumentation  will  be  used  to  provide  reliable  and 
adequate  data  for  systems  evaluation.  (5000.3,  C.l.e) 


MIL-H-46855B  states  that  human  factors  engineers  are  to 
be  involved  in  determining  the  role  of  man  in  the  system  and 
in  function  identification. 
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•  Estimates  of  Potential  Operator/Maintainer  Processing 
Capabilities  -  Plausible  human  roles  (e.g.,  operator, 
maintainer,  programmer,  decision  maker,  communicator, 
monitor)  in  the  system  shall  be  identified.  Estimates 
of  processing  capability  in  terms  of  load,  accuracy, 
rate  and  time  delay  shall  be  prepared  for  each  potential 
operator/maintainer  information  processing  function. 

These  estimates  shall  be  used  initially  in  determining 
allocation  of  functions  and  shall  later  be  refined  at 
appropriate  times  for  use  in  definition  of  operator/ 
maintainer  information  requirements  and  control,  display 
and  communication  requirements.  In  addition,  estimates 
shall  be  made  of  the  effects  on  these  capabilities  likely 
to  result  from  implementation  or  nonimplementation  of 
human  engineering  design  recommendations.  (3. 2. 1.1. 2) 

•  Starting  with  a  mission  analysis  developed  from  a  base¬ 
line  scenario,  the  functions  that  must  be  performed  by 
the  system  in  achieving  its  mission  objectives  shall  be 
identified  and  described.  (3. 1.1. a) 


Concept  Development  Phase 

The  need  for  integration  of  HF  R&D  in  all  aspects  of  system 
development  is  clearly  recognized  in  DOD  Directive  5000.1.  In 
addition,  the  Directive  recognizes  that  the  system  design  will 
influence  the  number  of  persons  and  the  skill  level  required  to 
operate  and  maintain  the  system. 

•  The  number  and  skill  levels  of  personnel  required 
and  human  engineering  factors  shall  be  included  as 
constraints  in  system  design.  The  integration  of 
the  human  element  and  system  shall  start  with  initial 
concept  studies  and  refined  as  the  system  program 
progresses  to  form  the  basis  for  personnel  selection 
and  training,  training  devices,  simulators,  and 
planning  related  to  human  factors  (5000.1,  IV.  W) 

•  Logistic  support  planning  including  reliability  and 
maintainability  shall  be  consistent  with  the  key 
program  decisions  and  phases  of  activity.  Alternative 
maintenance  concepts  shall  be  considered  during  the 
exploration  of  alternatives  to  identify  the  impact  on 
system  design  and  resources  (5000.1,  IV. V) 


•  Test  and  evaluation  shall  commence  as  early  as  possible. 
(5000.1,  IV. V) 


A  major  activity  for  HF  R&D  in  this  phase  is  function 
allocation.  MIL-H-46855B  states  that  this  activity  will  be  done: 


•  Defining  and  Allocating  System  Functions  -  The  functions 
that  must  be  performed  by  the  system  in  achieving  its 
objective (s)  within  specified  mission  environments  shall 
be  analyzed.  Human  engineering  principles  and  criteria 
shall  be  applied  to  specify  personnel-equipment/software 
performance  requirements  for  system  operation,  maintenance 
and  control  functions  and  to  allocate  system  functions  to 
(1)  automatic  operation/maintenance,  (2)  manual  operation/ 
maintenance,  or  (3)  some  combination  thereof.  Function 
allocation  is  an  iterative  process  achieving  the  level 
of  detail  appropriate  for  the  level  of  system  definition. 
(3. 2. 1.1) 


It  is  the  Program  Manager's  responsibility  to  develop  a 
good  plan  for  the  exploration  of  alternative  concepts.  The 
Program  Manager  will  need  man-machine  and  man-machine-environment 
HF  inputs  to  determine  which  man-machine  combinations  are  to  be 
effective  in  the  given  mission  environment  and  to  avoid  marginally 
effective  alternatives. 

•  Business  planning  should  emphasize  early  competitive 
exploration  of  alternatives  to  avoid  premature  commit¬ 
ment  to  solutions  that  may  prove  costly  and  marginally 
effective.  The  solicitation  for  proposed  solutions 
shall  be  in  terms  of  mission  needs  and  not  explicit 
system  characteristics  and  shall  provide  complete 
information  including  the  mission  task  and  the  operating 
environment  and  threat  to  enable  all  sources  to  respond 
fully  to  the  need.  (5000.2,  IV.E.l.e) 

•  The  Program  Manager  shall  systematically  and  progres¬ 
sively  explore  and  develop  alternative  system  concepts 
to  satisfy  the  approved  need.  (5000.1,  IV.D.l.b) 


Through  the  HF  and  other  inputs ,  the  Program  Manager  makes 
the  performance,  cost,  and  schedule  program  trade-off  decisions 
for  the  current  system.  The  Program  Manager  also  must  make 
provisions  to  incorporate  the  experience  gained  on  the  present 
system  to  become  part  of  a  data  base  for  future  system  decisions. 


•  Every  effort  shall  be  made  to  prevent  the  expenditure 
of  resources  to  achieve  unnecessary  performance  and 
schedule  requirements.  Meaningful  relationships  shall 
be  established  between  system  capability,  cost,  and 
schedule.  The  Program  Manager  shall  take  positiv  action 
to  continually  assess  program  risk  areas  and  to  make  or 
propose  trade-offs  in  performance,  cost,  and  schedule  to 
achieve  the  best  balance.  (5000.2,  IV. F. 6) 

•  A  realistic  work  breakdown  structure  shall  be  developed 
for  each  program  as  a  framework  for  planning  and  assign¬ 
ment  of  responsibilities,  control  and  reporting  progress, 
and  use  as  a  data  base  in  making  future  cost  estimates  of 
new  defense  systems.  (5000.2,  IV.F.ll) 


Directive  5000.3  has  provisions  for  developmental  or 
operational  testing  in  the  Concept  Development  Phase  if  the  PM 
feels  it  is  needed  to  choose  the  best  alternative.  HF  inputs 
are  needed  for  any  operational  test. 

•  During  the  Concept  Development  Phase  following 
Milestone  0,  adequate  DT&E  shall  be  accomplished 
when  appropriate  to  assist  in  selecting  preferred 
alternative  system  concepts.  (5000.3,  C.2.b) 

•  During  the  Concept  Development  Phase  following 
Milestone  0,  OT&E  will  be  conducted  when  appropriate 
to  assess  the  operational  impact  of  the  candidate 
technical  approaches  and  to  assist  in  selecting 
preferred  alternative  system  concepts.  (5000.3, 

C. 3. a (1) ) 


Directive  5000.3,  paragraph  3  calls  for  an  independent  test 
planner/evaluator.  Generally,  this  independent  evaluation  comes 
during  DT/OT  II,  but  could,  under  certain  circumstances,  begin  as 
early  as  DT/OT  I.  The  responsibilities  of  the  independent  test 
and  evaluation  agency  are: 

•  Participate  in  initial  program  planning  of  DT&E  to 
ascertain  what  portion  of  DT&E  will  contribute  to  the 
accomplishment  of  OT&E  objectives. 

•  Monitor  and  review  the  results  of  DT&E  as  necessary  to 
obtain  information  applicable  to  OT&E  objectives  and 
to  assess  the  readiness  of  the  system  for  operational 
testing. 
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•  Insure  that  the  various  phases  of  OT&E  are  scheduled  to 
allow  adequate  time  for  a  thorough  review,  analysis,  and 
reporting  of  test  results. 

•  Bring  directly  to  the  attention  of  its  Military  Service 
Chief  and  Service  Secretary,  or  Defense  Agency  Director, 
those  program  issues  which  impact  adversely  on  the 
accomplishment  of  adequate  OT&E. 

•  Provide  to  its  Service  Systems  Acquisition  Review 
Council  an  assessment  as  to  the  results  and  adequacy 
of  operational  testing  accomplished  to  date  and  the 
adequacy  of  operational  testing  planned  for  the  future 
to  support  the  Council's  recommendations. 

•  Provide  each  DOD  Component  within  its  immediate 
headquarters  staff  a  strong,  full-time,  focal  point 
organization  to  assist  the  independent  OT&E  field 
agency  to  keep  its  Military  Service  Chief  and  Service 
Secretary,  or  Defense  Agency  Director,  fully  informed 
as  to  OT&E  needs  and  accomplishments. 

•  Coordinate  test  planning  for  DT&E  and  OT&E  at  the  test 
design  stages  so  that  each  test  cycle  requires  minimum 
resources  and  yields  the  data  necessary  to  satisfy  the 
common  needs  of  the  materiel  developing  agency  and  the 
OT&E  agency.  The  purpose  of  the  coordination  is  to 
preclude  unnecessary  duplication.  In  general,  OT&E  will 
be  conducted  separately  from  DT&E;  however,  they  may  be 
combined  where  clearly  identified  and  significant  cost/ 
time  benefits  would  result,  or  where  separate  testing 
would  result  in  delay  involving  unacceptable  military 
risk  or  an  unacceptable  increase  in  the  acquisition 
cost  of  the  system.  When  combined  testing  is  conducted, 
the  necessary  test  conditions  and  test  data  required  by 
both  the  developing  agency  and  the  OT&E  agency  must  be 
realized.  Therefore,  the  developing  agency,  through 
coordination  with  the  OT&E  agency,  must  insure  that  the 
combined  test  is  so  planned  and  executed  as  to  provide 
the  necessary  operational  test  information  and  that  the 
OT&E  agency  is  afforded  the  opportunity  to  participate 
actively  in  the  test  design  and  execution.  The  OT&E 
agency  shall  provide  a  separate  evaluation  of  the 
resultant  operational  test  information.  As  a  normal 
practice  the  operational  tests  supporting  a  production 
decision  will  be  conducted  independently  by  the  OT&E 
agency. 
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•  Structure  acquisition  programs  so  that  operational 
testing  is  commenced  as  early  as  possible  in  the 
development  cycle.  As  a  minimum,  an  initial  phase 
of  operational  test  and  evaluation  (IOT&E)  will  be 
accomplished  prior  to  the  Production  and  Deployment 
Decision  Milestone  3.  Such  IOT&E  shall  be  adequate 
to  provide  a  valid  estimate  of  expected  system  oper¬ 
ational  effectiveness  and  operational  suitability. 
Preproduction  prototypes  or  full-scale  development 
items  will  be  employed  for  IOT&E  if  they  are  reasonably 
representative  of  the  expected  production  items  and 
will  allow  a  valid  estimate  to  be  made  of  expected 
system  operational  effectiveness  and  operational 
suitability;  otherwise,  pilot  production  items  will 
be  employed  for  IOT&E. 


Decision  Coordinating  Paper 

After  testing  and  analyzing  the  alternative  concepts,  the 

*  Decision  Coordinating  Paper  (DCP)  should  identify  the  critical 
issues  and  areas  of  risk  to  be  addressed  by  test  and  evaluation 
(Directive  5000.3,  C.lO.a).  HF  R&D  is  needed  to  address  the 
human,  human-machine,  and  human-machine-mission  issues  to  resolve 

»  any  debilitating  system  aspects.  HF  R&D  is  also  needed  to  pro¬ 

vide  human,  human-machine,  and  human-machine-mission  measures  of 
effectiveness  related  to  satisfaction  of  mission  need  (Directive 
5000.3,  C.lO.a).  In  summary,  the  Directives  call  for  the 

♦  following  issues,  all  of  which  need  HF  R&D  analysis: 

•  Description  of  the  alternative  programs,  including 
anticipated  performance  information.  (5000.2, 

Enclosure  2,  II. C) 

•  Areas  of  program  uncertainty  (excluding  technical  risks) 
and  the  probable  impact.  (5000.2,  Enclosure  2,  III.G) 

•  Program  issues,  including  their  assessment  (5000.2, 
Enclosure  2,  II.N) 

•  DCPs  prepared  for  Milestone  1  shall  contain  program 
management  constraints  for  selected  program  factors 
for  each  alternative  as  the  basis  for  continuing  the 
competitive  demonstration  effort  for  the  particular 
alternative.  (5000.2,  Enclosure  2,  II. K) 
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•  Each  DCP  prepared  for  Milestones  1  and  2  shall  contain 
a  Technology  Assessment  Annex  (TAA)  that  will  identify 
any  area  of  technological  risk  remaining  in  the  program 
and  describe  plans  for  addressing  these  risks.  (5000.1, 
Enclosure  2,  III.H) 

•  DCPs  prepared  for  Milestones  2  and  3  shall  contain  firm 
program  schedules,  cost,  and  performance  information. 
Program  thresholds  shall  be  established  for  selected 
performance,  cost,  and  schedule  factors  representing 
acceptable,  projected  variances  at  program  completion 
and  fiscal  year  thresholds  for  the  same  cost  and  schedule 
factors  to  represent  acceptable  variances  at  the  end  of 
each  fiscal  year.  (5000.1,  Enclosure  2,  III.L) 

•  Test  and  evaluation  planning  and  status.  (5000.2, 
Enclosure  2,  II. M) 


At  the  next  decision  point,  Milestone  1,  the  issues  to  be 
addressed  by  (S)SARC,  DSARC ,  and  the  Secretary  of  Defense  are 
contained  in  Directive  5000.2,  Enclosure  2.  The  following 
excerpts  are  HF-related. 

•  The  alternative  system  design  concepts  adequately  reflect 
the  technology  base  and  provide  an  acceptable  competitive 
environment.  (IV. A. 3) 

•  The  alternatives  recommended  for  competitive  system 
demonstration  meet  the  mission  element  needs.  (IV. A. 5) 

•  The  established  program  constraints  remain  valid. 

(IV. A. 6) 

•  The  projected  resource  investment  for  the  selected 
alternatives  and  other  characteristics  related  to  the 
alternatives  are  consistent  with  the  stated  constraints. 
(IV. A. 7) 

•  Operational  and  logistical  considerations  are  adequate. 
(IV. A. 8) 

•  NATO  standardization  and  interoperability  requirements 
have  been  adequately  considered.  (IV. A. 14) 

•  Risk  and  uncertainty  areas  are  identified  and  adequately 
treated  in  the  planning.  (IV. A. 15) 


•  Environmental  considerations  are  adequate  (IV. A. 16) 

•  Planning  and  schedules  for  preparation  of  the  Test  and 
Evaluation  Master  Plan  (TEMP)  are  adequate.  (IV. A. 17) 

Demonstration  and 
Validation  Phase 

Competitive  systems  demonstrations  are  conducted  to  validate 
the  design  concepts  and  to  provide  a  basis  for  selection  of  a 
system  for  full-scale  development  and  subsequent  production. 
Directive  5000.2  (IV. F. 8)  states  that  the  demonstrations  should 
be  conducted  with  full-scale  prototypes  in  realistic  operating 
environments  when  feasible  and  practical.  When  demonstrations  at 
the  system  level  are  determined  not  to  be  feasible  and  practical, 
competitive  prototypes  demonstrations  of  critical  subsystems 
shall  be  considered  in  the  same  manner  as  systems.  Human  factors 
test  and  evaluation  are  needed  in  this  system  development  phase 
to  assure  that  the  proposed  man-machine-mission  interfaces  are 
feasible  and  produce  good  performance  in  a  realistic  operating 
environment.  The  test  and  evaluations  provide  inputs  for  the 
next  phase,  as  system  design  should  not  yet  be  firm.  Specifically, 
Directive  5000.1  (IV. Q)  states  that: 

•  Performance,  cost,  and  schedule  estimates  shall  not  be 
formalized  or  considered  firm  prior  to  the  Milestone  2 
decision  since  systems  are  not  adequately  defined  and 
the  value  for  these  system  parameters  remain  uncertain 
during  the  early  phases  of  the  system  acquisition 
process . 

The  Test  and  Evaluation  Master  Plan  (TEMP)  should  specify 
how  the  competitive  systems  are  to  be  evaluated.  All  test  and 
evaluation  aspects,  including  human  factors,  should  be  contained 
in  the  TEMP.  The  DOD  Component  is  responsible  for  developing 
this  plan. 

•  The  DOD  Component  shall  prepare,  as  early  as  possible  in 
the  acquisition  process,  an  initial  version  of  the  overall 
test  and  evaluation  plan.  This  broad  plan  should  identify 
and  integrate  objectives,  responsibilities,  resources, 
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and  schedules  for  all  T&E  to  be  accomplished  prior  to 
the  subsequent  key  decision  points.  The  TEMP  will  be 
kept  current  by  the  DOD  Component.  (5000.3,  C.8) 


Task  analysis  is  a  major  activity  in  this  phase.  MIL-H- 
46855B  states  the  purpose  and  need  of  these  analyses: 

•  The  analyses  shall  provide  one  of  the  bases  for  making 
design  decisions;  e.g.,  determining,  to  the  extent 
practicable,  before  hardware  fabrication,  whether  system 
performance  requirements  can  be  met  by  combinations  of 
anticipated  equipment,  software,  and  personnel,  and 
assuring  that  human  performance  requirements  do  not 
exceed  human  capabilities.  These  analyses  shall  also 

be  used  as  basic  information  for  developing  preliminary 
manning  levels;  equipment  procedures;  skill,  training 
and  communication  requirements;  and  as  Logistic  Support 
Analysis  inputs,  as  applicable.  Those  gross  tasks 
identified  during  human  engineering  analysis  which 
are  related  to  end  items  of  equipment  to  be  operated 
or  maintained  by  personnel  and  which  require  critical 
(see  6.2.1)  human  performance,  reflect  possible  unsafe 
practices  or  are  subject  to  promising  improvements  in 
operating  efficiency  shall  be  further  analyzed,  with 
the  approval  of  the  procuring  activity.  (3. 2. 1.3.1) 

•  Each  task  is  analyzed  to  determine  the  human  performance- 
parameters,  the  system/equipment/software  capabilities, 
and  the  tactical/environmental  conditions  under  which 
the  tasks  are  conducted.  Task  parameters  shall  be 
quantified,  where  possible,  and  in  a  form  permitting 
effectiveness  studies  of  the  crew-equipment/software 
interfaces  in  relation  to  the  total  system  operation. 

The  identification  of  human  engineering  high  risk  areas 
shall  be  initiated  as  part  of  the  analysis.  (3. 1.1. a) 

•  Further  analysis  of  critical  tasks  shall  identify  the: 

(I)  information  required  by  operator/maintainer , 
including  cues  for  task  initiation;  (2)  information 
available  to  operator/maintainer;  (3)  evaluation  process; 
(4)  decision  reached  after  evaluation;  (5)  action  taken; 
(6)  body  movements  required  by  action  taken,  (7)  work¬ 
space  envelope  required  by  action  taken;  (8)  workspace 
available;  (9)  location  and  condition  of  the  work 
environment;  (10)  frequency  and  tolerances  of  action; 

(II)  time  base;  (12)  feedback  informing  operator/ 
maintainer  of  the  adequacy  of  actions  taken;  (13)  tools 
and  equipment  required;  (14)  number  of  personnel  required; 
(15)  job  aids  or  references  required;  (16)  communications 


required,  including  type  of  communication;  (17)  special 
hazards  involved;  (18)  operator  interaction  where  more 
than  one  crew  member  is  involved;  (19)  operational 
limits  of  personnel  (performance) ;  and  (20)  operational 
limits  of  machine  and  software.  The  analysis  shall  be 
performed  for  all  affected  missions  and  phases  including 
degraded  modes  of  operation.  (3. 2. 1.3. 2) 


At  the  present  time,  the  Triservice  Human  Factors  Test  and 
Evaluation  sub-group  of  the  Department  of  Defense  Human  Factors 
Engineering  Technical  Advisory  Group  (TAG)  has  begun  development 
of  a  coordinated  means  for  task  analysis  development.  While  it 
was  unavailable  for  review  in  this  technical  report,  it  is 
understood  that  it  will  supersede  the  task  analysis  requirements 
as  specified  in  MIL-H-46855. 

MIL-STD-1472B 

The  purpose  of  MIL-STD-1472B  is  to  establish  human  engi¬ 
neering  design  criteria  for  the  design  and  development  of  military 
systems  and  facilities.  The  concerns  of  MIL-STD-1472B  range  from 
function  allocation  to  safety,  control-display  relationships, 
visual  displays,  audio  displays,  controls,  labeling,  anthropometry, 
workplace  design,  environment,  maintainability  design,  equipment 
for  remote  handling,  and  aerospace  vehicle  compartments  design 
requirements.  While  1472B  may  be  useful  in  considering  the 
ramifications  of  options  in  the  Mission  Analysis  and  Concept 
Development  Phase,  the  primary  use  of  1472B  is  in  insuring 
contractors'  adherence  to  the  human  factors  R&D  requirements  in 
the  Demonstration  and  Validation  Phase,  Full-Scale  Development 
Phase,  Production  Phase,  and  for  improvements  and  retrofits  after 
the  Production  and  Deployment  Phase. 


Developmental  testing  is  the  proper  time  to  test  the  man- 
machine  interface.  The  purposes  of  developmental  test  and 
evaluation  are  to  identify  the  preferred  technical  approach 


(Directive  5000.3,  C.2C)  and  to  assist  the  engineering  design 
and  development  process  and  verify  attainment  of  technical 
performance  specifications  and  objectives  (Directive  5000.3, 
C.2).  Developmental  test  and  evaluation  includes  testing  of 
components,  subsystems,  hardware/software  integration,  and 
prototypes  (Directive  5000.3,  C.2. a).  To  be  sure  that  the 
new  system  will  not  negatively  interact  with  existing  or 
planned  equipment  and  systems,  it  is  necessary  to  test  the 
compatibility  and  interoperability  of  interfacing  equipment. 


Operational  testing  should  occur  in  a  realistic  environment 
and  is  the  time  to  test  the  human-machine-mission  interface. 
Directive  5000.2  (C.3.a(2))  states  that: 

•  During  the  Competitive  Systems  Demonstration  Phase, 

OT&E  will  be  conducted,  as  necessary,  in  as  realistic 
an  environment  as  possible  in  order  to  examine  the 
operational  aspects  of  selected  technical  approaches, 
and  to  provide  information  relative  to  projected 
operational  effectiveness  and  suitability  of  the 
candidate  systems. 


Directive  5000. 3  (c. 3)  is  more  complete  about  the  issues  in 
operational  test  and  evaluation. 

•  OT&E  is  that  test  and  evaluation  conducted  to  estimate 
a  system's  operational  effectiveness  (including 
survivability /vulnerability)  and  operational  suitability 
(including  availability,  compatibility,  transportability, 
interoperability,  reliability,  maintainability,  safety, 
human  factors,  logistic  supportability ,  and  training 
requirements),  as  well  as  the  need  for  any  modifications. 
In  addition,  OT&E  provides  information  on  organization, 
personnel  requirements,  doctrine,  and  tactics.  It  may 
also  provide  data  to  support  or  verify  material  in  oper¬ 
ating  instructions,  software  documentation,  publications, 
and  handbooks.  OT&E  will  be  accomplished  by  operational 
and  support  personnel  of  the  type  and  qualifications  of 
those  expected  to  use  and  maintain  the  system  when 
deployed,  and  will  be  conducted  in  as  realistic  an 
operating  environment  as  possible,  including  enemy 
countermeasures.  It  will  address  operational  perfor¬ 
mance  of  component  systems,  including  new  or  improved 


components  and  their  interaction  with  related  existing 
systems.  Costs  directly  related  to  operational  testing 
in  a  program  shall  be  planned,  programmed,  and  budgeted 
by  the  cognizant  DOD  component. 


Following  developmental  and  operational  test  and  evaluation, 
the  DCP  is  updated  to  give : 

•  The  results  of  T&E  accomplished  to  date;  and  updated 
statement  of  critical  issues,  test  objectives,  and 
areas  of  risk  needing  further  assessment,  a  summary 
of  performance  criteria  goals  and  thresholds,  and 
an  overview  of  test  plans,  milestones,  and  program 
interrelationships.  The  cognizant  Component  shall 
make  available  supporting  details  of  test  plans  and 
test  results  as  requested  by  the  DDTE  or  ASD  (PA&E) . 
(5000.3,  C.lO.b) 


Excerpts  from  Directive  5000.2  give  the  human  factors 
Milestone  2  issues  to  be  addressed. 

•  System  tradeoffs  have  produced  the  most  effective 
balance  in  cost,  performance,  and  schedule  including 
operational  and  logistical  considerations.  (IV. B. 5) 

•  Uncertainties  and  risks  have  been  identified  and  are 
acceptable;  planning  to  resolve  the  remaining  uncer¬ 
tainties  and  risks  is  adequate.  Realistic  fall-back 
actions  and  alternatives  have  been  established. 

(IV. B. 6) 

•  Cost,  performance,  and  schedule  estimates  and  related 
thresholds  have  been  thoroughly  reviewed  and  are  well 
defined  and  consistent  with  risks  involved.  These 
values  shall  be  established  as  firm  estimates. 

(IV. B. 10) 

•  Demonstration  and  validation  testing  and  evaluations 
have  been  completed  and  results  support  the  recommen¬ 
dations.  (IV. B. 13) 

•  Requirements  have  been  established  for  long-lead 
procurement  items  and  initial  limited  production  to 
support  operational  test  and  evaluation  needs,  for 
the  verification  of  production  engineering  and  design 
maturity,  and  to  establish  the  production  base. 

(IV. B. 16) 
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•  The  system  selected  meets  the  mission  element  needs, 
is  cost-effective,  and  is  acceptable  within  stated 
constraints.  (IV. B. 2) 

•  NATO  standardization  and  interoperabolity  requirements 
are  satisfied.  (IV. B. 3) 

•  The  Test  and  Evaluation  Master  Plan  (TEMP)  identifies 
and  integrates  the  testing  and  evaluation  to  be 
accomplished  prior  to  the  Milestone  2  and  3  program 
decision  points.  (IV.B.17) 

•  Requisites  for  the  Milestone  3  production  and  deployment 
decision  including  operational  and  logistical  support 
have  been  established.  (IV.B.18) 


Full-Scale  Development  Phase 

By  the  end  of  this  phase,  the  hardware  configurations  and 
the  subsystem  structures  become  firm  unless  design  errors  are 
found. 

•  Subsystems  selected  for  use  in  a  system  acquisition 
program  shall  not  be  fully  developed  until  the  system 
program  has  been  approved  for  full-scale  engineering 
development.  (5000.2,  IV. F. 3) 


Because  the  hardware  configuration  is  becoming  firm,  it  is 
important  that  the  contractors  have  human  factors  personnel  to 
analyze  and  propose  the  proper  man-machine  interface.  Likewise, 
it  is  important  that  the  contractor's  proposals  have  design 
requirements  that  fulfill  the  human  factors  specifications, 
e.g.,  MIL-STD-1472B. 

The  degree  of  use  of  human  factors  data,  methodology,  and 
expertise  in  the  developing  system  will  be  only  as  good  as  that 
required  of  the  contractors  by  DOD. 

•  Contractors  shall  be  required  to  submit  firm  proposals 
for  full-scale  engineering  development  and  initial 
production  upon  completion  of  the  competitive  systems 
demonstration  and  shall  be  provided  with  the  factors, 
criteria,  and  conditions  to  be  used  by  the  DOD  in  the 
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evaluation  and  selection  of  a  system  for  full-scale 
engineering  development.  Specifications  and  standards 
and  a  contract  data  list  shall  be  identified  and  tailored 
by  the  contractors  for  application  to  the  system  proposal 
for  full-scale  development  on  the  basis  of  the  competitive 
systems  demonstration  results.  (5000.2,  IV. F. 8) 


Proper  human  factors  inputs  should  make  the  system  more  cost 
effective.  The  results  of  human  factors  analyses  should  be  used 
in  calculating  operating  maintenance  life  cycle  costs.  Decisions 
resulting  from  human  factors  analyses  can  affect  needed  skill 
levels  to  operate  the  system,  amount  of  training,  type  of  documen¬ 
tation,  operating  and  maintenance  error  rates,  and  the  maintenance 
philosophy. 

•  Costs  of  acquisition  and  ownership  shall  be  established 
as  separate  cost  elements  and  translated  into  firm 
design-to-cost  and  life  cycle  cost  requirements  for  the 
system  selected  for  full-scale  engineering  development. 
System  program  actions  shall  be  evaluated  against  these 
requirements  with  the  same  rigor  as  the  evaluation  of 
technical  requirements.  (5000.2,  IV. F. 7) 


Logistical  planning  also  becomes  firm  in  this  phase: 

•  Detailed  logistics  planning  shall  be  initiated  with  full- 
scale  engineering  development  and  firm  requirements 
established  early  in  the  phase.  The  adequacy  of  logistics 
plans,  and  resources  to  meet  readiness  objectives,  will 
be  reviewed  as  part  of  the  Milestone  3  production  decision 
(5000.1,  IV. V) 


The  human  factors  issues  should  be  resolved  during  this  phase 

•  During  the  Full-Scale  Engineering  Development  Phase  prior 
to  the  Production  and  Deployment  Decision  (Milestone  3) , 
the  DT&E  accomplished  shall  be  adequate  to  insure:  that 
engineering  is  reasonably  complete;  that  all  significant 
design  problems  (including  survivability /vulnerability , 
availability,  producibility ,  compatibility,  transport¬ 
ability,  interoperability,  reliability,  maintainability, 
safety,  human  factors,  electromagnetic  compatibility,  and 
logistic  supportability)  have  been  identified  and  that 
solutions  to  these  problems  are  in  hand.  (5000.3,  C.2.d) 
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Human  factors  test  and  evaluation  plays  a  major  role  in 
full-scale  development  operational  testing. 

•  An  estimate  of  military  utility  and  of  operational 
effectiveness  and  operational  suitability,  including 
logistics  support  requirements,  shall  be  made  prior 
to  large-scale  production  commitments.  The  most 
realistic  test  environment  possible  and  an  acceptable 
representation  of  the  future  operational  system  will 
be  used  in  the  testing..  (5000.1,  IV. U) 


While  the  directives  call  for  operational  testing  in  the 
prior  phase,  the  first  complete  and  major  operational  test  is 
intended  to  occur  during  the  Full-Scale  Development  Phase. 

•  OT&E  will  normally  be  conducted  in  phases,  each  keyed 

to  an  appropriate  decision  point;  OT&E  conducted  prior 
to  the  Milestone  3  decision  is  designated  Initial 
Operational  Test  and  Evaluation  (IOT&E) .  (5000.3, 

C.  3. a) 

•  During  the  Full-Scale  Engineering  Development  Phase, 
following  the  Milestone  2  decision,  adequate  OT&E 
will  be  accomplished  to  provide  a  valid  estimate  of 
operational  effectiveness  and  suitability.  (5000.3, 

C. 3. a (3)  ) 


Those  human  factors  issues  to  be  addressed  at  Milestone  3 
have  been  taken  from  Directive  5000.2. 

•  The  development  has  progressed  satisfactorily  and  the 
initial  operational  test  and  evaluation  results  support 
a  decision  to  proceed  with  production  and  deployment. 

(IV. C. 2) 

•  Schedule  and  cost  estimates  are  realistic  and  acceptable 
including  support  and  operating  costs.  (IV.C.5) 

•  Tradeoffs  have  been  made  to  balance  cost,  schedule,  and 
performance  effectively.  (IV. C. 8) 

•  Major  problems  are  identified  and  satisfactorily 
resolved.  (IV.C.13) 

•  NATO  standardization  and  interoperability  requirements 
have  been  satisfied.  (IV.C.14) 
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•  Planning  for  deployment  is  adequate,  including  manpower 
and  training  logistics  readiness;  and  operational 
considerations,  including  integration  with  existing 
operational  systems.  (IV.C.16) 

•  Assessment  of  support  subsystems  to  meet  needs  of  initial 
operational  units  and  planning  to  meet  any  deficiencies. 
(IV.C.17) 


Production  and  Deployment  Phase 

Once  a  system  goes  into  production,  there  are  opportunities 
for  inputs  and  changes  in  the  system  configuration  on  an 
as-needed  basis. 

•  Following  a  Milestone  3  decision,  the  DOD  Component  Head 
shall  make  quarterly  reports  to  the  Secretary  of  Defense 
on  key  program  issues.  The  DOD  Component  shall  keep  the 
Defense  Acquisition  Executive  and  the  OSD  staff  informed 
on  key  program  actions  as  the  program  progresses. 

(5000.1,  IV. D . 4 .b) 

•  In  case  of  DCP  revisions  and  DSARC  or  (S)SARC  reviews 
subsequent  to  the  Production  and  Deployment  Decision  an 
updated  assessment  of  test  results,  plans,  objectives, 
and  schedules  for  additional  test  and  evaluation  will 
be  provided.  (5000.3,  C.lO.d) 


While  developmental  and  operational  testing  occur  on  an 
as-needed  basis,  production  acceptance  test  and  evaluation 
(PAT&E)  is  required  during  this  phase. 

•  During  the  Production  and  Dep-  Tient  Phase  following  the 
Milestone  3  decision,  DT&E  will  be  an  integral  part  of 
the  development,  acceptance,  and  introduction  of; 

(1)  Product  improvements  into  the  produced  system. 

(2)  Operational  characteristic  modifications  to  meet 
identified  threat  changes. 

(3)  Changes  to  reduce  system  life  cycle  costs.  (5000.3, 
C. 2 . e) 
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•  Subsequent  to  the  Milestone  3  decision,  OT&E  will  be 
continued  as  necessary  to  refine  estimates  of  operational 
effectiveness,  to  evaluate  changes,  and  to  re-evaluate 
the  system  to  ascertain  whether  it  continues  to  meet 
operational  needs  and  retains  its  effectiveness  in  a  new 
environment  or  against  the  current  or  projected  new 
threat.  (5000.3,  C.3.a(4)) 

•  PAT&E  is  test  and  evaluation  of  production  items  to 
demonstrate  that  items  procured  fulfill  the  requirements 
and  specifications  of  the  procuring  contract  or  agree¬ 
ments.  It  is  the  responsibility  of  each  DOD  Component 
to  accomplish  the  necessary  PAT&E  throughout  the 
Production  and  Development  Phase  of  the  acquisition 
process.  (5000.3,  C.7) 


In  summary,  the  DOD  Directives,  MIL-H-46855B,  and  MIL-STD- 
1472B  state  the  need  for  human  factors  R&D  in  all  system  acquisition 
phases.  These  human  factors  R&D  analyses  are  conducted  to  insure 
that  the  most  operationally  and  cost-effective  system  is  developed 
within  the  constraints  of  the  program. 


t 


♦ 


♦ 


t 
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Human  Factors  R&D:  Service  Requirements 


Service  regulations  and  instructions  that  document  human 
factors  activities,  albeit  of  a  general  nature,  provide  the 
framework  for  activities  throughout  the  military  system  acqui¬ 
sition  process.  To  the  extent  possible,  requirements  for  human 
factors  will  be  related  to  the  overall  system  acquisition  cycle 
as  well  as  to  specific  phases  within  it.  The  intent  is  to  show 
the  degree  to  which  the  services  are  concerned  with  the  inte¬ 
gration  of  human  factors  R&D  into  the  process  by  which  new 
military  systems  are  acquired  and  developed.  The  three  pertinent 
documents  are: 

1.  Department  of  the  Air  Force,  AF  Regulation  800-15. 

Vi.-c*:  Factors  Engineering  and  Management . 

2.  Department  of  the  Army,  Army  Regulation  602-1. 

Human  Factors  Engineering  Program. 

3.  Department  of  the  Navy,  NAVMATINST  3900.9. 

Human  Factors. 

Generally,  topics  covered  in  these  documents  have  included: 

•  definitions  of  human  factors  and/or  human  factors 
engineering , 

•  broadly  based  descriptions  of  activities  and  functions 
of  human  factors, 

•  designated  responsibilities  for  human  factors, 

•  extent  of  involvement  of  human  factors  with  system  life 
cycles ,  and 

•  coordination  of  requirements  with  documentation  such  as 
military  specifications  and  standards. 
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The  intent  has  been  to  provide  generic  data  requirements  and  not 
to  restrain  human  factors  developments  and  innovations.  In  fact, 
innovation  in  human  factors  methodology  application  is  encouraged. 
The  following  quote  from  Army  Regulation  602-1,  illustrates  this 
point : 

Human  Engineering  standards  which  embody  well- 
tested  human  factors  principles  or  prescribe 
design  standardization  to  minimize  cross-training 
or  relearning  problems  are  appropriate  as 
guidelines  ....  These  standards  should  not 
preclude  design  approaches  which  can  lead  to 
improved  performance  of  the  personnel-materiel 
system. 

On  the  other  hand,  informal  requirements  (e.g.  ,  handbooks, 
guidebooks,  and  manuals)  for  human  factors  espoused  by  the 
different  services  have  been  more  disposed  to  discuss  methodology 
applications.  Whatever  human  factor  elements  are  included  and 
whatever  the  depth  with  which  these  have  been  discussed,  the 
overall  emphasis  seems  to  be  the  placement  of  human  factors  in 
system  development  as  a  total  systems  view. 


Overall  Military  System  Acquisition 
Human  Factors  R&D  Requirements 

Since  the  service  documents  are  more  directly  oriented  to 
the  definition  and  activity  requirements  for  human  factors,  and 
less  concerned  with  specific  functions  as  they  relate  to  system 
development,  most  statements  involving  human  factors  R&D  in 
development  of  systems  cover  the  overall  process.  This  results 
in  a  greater  level  of  general  detail.  The  intent  is  to  provide 
few  constraints  upon  use  of  human  factors  methods  by  allowing  for 
greater  flexibility. 
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Greatest  detail  with  regard  to  the  overall  process  is  pro¬ 
vided  in  the  Army  regulation.  As  well  as  detailing  the  events 
themselves,  this  document  provides  information  about  when  human 
factors  requirements  should  be  invoked. 


In  addition  to  detailing  the  kinds  of  events  relevant  to 
the  overall  process  of  human  factors  R&D  in  system  development, 
the  Army  regulation  provides  information  about  when  human  factors 
requirements  should  be  invoked. 

During  all  phases  of  the  life  cycle  for  materiel 
systems,  accepted  principles  of  HFE  will  be  used 
to  integrate  material  development  with  personnel 
resources  and  will  have  an  associate  priority 
with  all  other  systems  characteristics.  Personnel 
implications  will  be  considered  throughout  all 
development  activities.  Human  factors  studies 
or  behavioral  research  will  be  initiated  when  gaps 
in  HFE  data  base  exist  and  where  novel  human  factors 
problems  are  identified  in  Army  development  programs. 

During  all  phases  of  systems  development,  the 
interactive  effect  of  system  concept,  hardware 
design,  software  design,  personnel  performance 
requirements,  and  training  requirements  must  be 
recognized. 


The  Department  of  the  Navy  instruction  provides  requirements 
which  are  of  primary  concern  to  the  Navy  in  systems  development. 

In  fact,  the  Navy  considers  it  necessary  for  the  "human  element" 
to  proceed  through  developmental  steps  similar  to  that  of  hardware. 

Human  factors  engineering  is  concerned  with  the 
analysis  and  design  of  systems  to  achieve  operability 
and  supportability  with  the  human  operator/maintainer 
in  the  system.  The  effort  begins  with  the  perfor¬ 
mance  requirements  analysis  and  includes  information 
flow  analysis,  function  allocation,  maintenance/ 
maintainability  analysis,  equipment  design,  crew 
station  design,  test,  and  evaluation. 

The  Naval  Materiel  Command  human  factors  policy 
requires  that  the  human  element  of  Navy  systems 
shall  undergo  the  same  development ,  test ,  and 


evaluation  steps  as  equipment  elements  of  the  same 

system.  This  requires  integration  of  appropriate 
human  factors  information  into  design  and  its  use 
in  all  major  management  and/or  technical  decisions 
and  documents. 


The  Department  of  the  Air  Force  regulation  attempts  to 
legitimize  human  factors  within  the  engineering  context  by 
defining  it  as  having  a  shared  responsibility  for  system  devel¬ 
opment  through  the  system  engineering  approach.  In  addition, 
it  recognizes  the  individual  nature  of  each  system  and  the  manner 
by  which  human  factors  R&D  must  adapt  to  this  feature. 

HFE  is  a  part  of  the  mainstream  engineering  effort 
throughout  the  system  life  cycle.  It  is  that 
component  of  systems  engineering  which  seeks 
to  optimize  the  system  by  integrating  the  human 
performance  necessary  to  operate ,  maintain ,  support , 
and  control  the  system  in  its  intended  operational 
environment . 

The  consideration  of  HFE  requirements  must  begin 
with  the  inception  of  the  system  or  equipment  life 
cycle.  The  scope  of  this  effort  depends  on  the 
nature  and  the  type  of  system  or  equipment  program, 
and  must  be  tailored  to  meet  specific  program 
objectives. 

The  policy  stated  in  this  regulation  will  be 
adapted  to  meet  the  unique  requirements  of  each 
system  or  project,  considering  the  specific  phase 
of  the  system  life  cycle  at  hand,  the  scope  of 
the  system  or  project  in  work,  and  the  special 
management  needs  of  the  total  program.  The 
responsibility  for  HFE  begins  with  the  inception 
of  the  system  or  project  and  continues  throughout 
the  life  cycle  of  each  system  or  project. 


Service  Requirements  for  Human 
Factors  R&D  in  Mission  Analysis 

Less  often  considered  than  the  rest  of  the  phases,  mission 
analysis  requirements  are  less  clearly  defined  by  the  service 
documents.  The  Army  regulation  approaches  the  topic  by  including 
the  elements  deemed  pertinent  to  description  of  a  system's 
mission. 


It  is  important  that  decisions  regarding  mission, 
doctrine,  basis  of  issue,  unit  organization  and 
manning,  personnel  selection  and  training,  and 
technical  or  training  publications  all  be  made 
from  a  coordinated  data  base  including  human 
factors  engineering. 


On  the  other  hand,  the  Air  Force  refers  to  a  concept  often 
applied  to  mission  analyses  but  does  not  refer  to  the  phase  at 
all. 


Man's  role  in  the  system  is  defined  in  order 
to  optimize  his  performance  in  relation  to 
that  specific  system. 


Service  Requirements  for  Human 
Factors  in  Concept  Development 

The  services  recognize  the  importance  of  critical  human 
factors  events  which  must  be  prevalent  in  the  Concept  Development 
Phase.  Numerous  varieties  of  studies  are  referred  to  which  are 
of  direct  relevance  to  events  that  follow  in  later  phases. 


The  Army  regulation  delineates  specific  portions  of  human 
factors  R&D  which  must  be  initiated  during  this  phase.  In  addi¬ 
tion,  it  provides  the  logic  for  developing  concept  development 
data  to  be  used  as  system  development  objectives  for  later  stages. 

HFE  in  the  Conceptual  Phase.  HFE  will  be  initiated 
in  the  conceptual  phase  of  the  system's  life  cycle. 

During  the  conceptual  phase,  HFE,  personnel  plans, 
and  training  considerations  will  be  integrated 
into  the  technical  and  management  plans.  HFE  data 
developed  during  the  conceptual  phase  will  be  con¬ 
sidered  during  this  phase  in  determining  projected 
personnel  requirements  and  in  planning  development 
of  personnel-support  programs  and  training  programs. 

In  the  case  of  materiel  with  a  very  strong  human 
interface,  it  is  critically  important  to  develop 
HFE  data  in  the  conceptual  phase  (RDTE  category  6.2 
and  6.3a)  sufficient  to  provide  a  basis  for  thorough 
design  and  evaluation  during  Systems  Oriented 
Advanced  Development  (RDTE  Category  6.3b). 
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The  Navy  instruction  states  point-blank  the  requirement  for 
human  factors  in  concept  development,  since  it  is  such  a  brief 
document. 

As  a  minimum  this  will  involve  human  factors  inputs 
to  concept  formulation. 


The  Department  of  the  Air  Force  points  out  a  requirement  for 
human  factors  R&D  in  concept  development.  Rather  than  stopping 
there,  the  document  lists  the  considerations  that  human  factors 
studies  must  help  address.  In  addition,  they  reinforce  the  life 
cycle  nature  of  concept  development  activities,  and,  by  implica¬ 
tion,  how  human  factors  R&D  products  must  be  continually  refined 
throughout  the  system  development  process. 

HFE  must  be  an  integral  part  of  R&D  conceptual 
study  efforts  .... 

Adequate  man-machine  analyses  and  trade-off  studies 
are  accomplished  beginning  with  the  conceptual 
phase  and  as  appropriate  throughout  the  system 
life  cycle.  These  studies  must  consider  life 
cycle  costs,  system  performance  requirements 
and  complexity,  and  the  capability  of  available 
personnel  to  perform  the  intended  function. 


Service  Requirements  for  Human 
Factors  R&D  in  the  Demonstration 
and  Validation  Phase 

System  demonstration  and  validation  is  the  phase  whereby  the 
services  require  the  first  detailed  human  factors  analysis.  The 
documentation  links  directly  the  design/development  of  the  system 
prototype  and  the  availability  of  human  factors  data. 

The  Army  regulation  points  out  that  the  Demonstration  and 
Validation  Phase  offers  a  good  opportunity  for  human  factors 
contributions  to  design,  since  the  system  is  in  prototype  devel¬ 
opment  and  the  design  has  not  been  firmly  set.  In  addition, 
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the  document  stresses  the  development  of  human  factors  items 
(e.g.,  training  and  training  device  plans,  etc.)  concomitant 
with  other  research  in  this  phase. 

HFE  in  the  Validation  Phase.  Systems  Oriented 
Advanced  Development  (RDT&E  Category  6.3b)  is  the 
preferred  state  of  development  for  the  completion 
of  detailed  HFE.  At  this  stage,  prototypes 
representing  the  first  concrete  expression  of 
the  concept  are  available  for  test  and  evaluation, 
yet  the  design  has  not  been  frozen.  As  a  result, 
sufficient  information  concerning  the  proposed 
design  is  available  to  support  a  detailed  HFE 
analysis  while  this  design  is  still  sufficiently 
flexible  to  accommodate  any  change  resulting 
from  the  recommendations  of  this  analysis.  The 
requirement  to  initiate  an  HFE  analysis  as  a  part 
of  6.3b  will  be  documented  in  the  Letter  of 
Agreement  (LOA) .  It  will  include  the  designation 
of  the  agency  with  recognized  HFE  expertise 
responsible  for  conducting  and  reviewing  the  HFE 
analysis.  Results  of  the  analysis  will  include: 

(1)  Tentative  identification,  allocation,  and 
sequencing  of  operator  and  maintenance  tasks 
to  develop  a  concept  of  the  training  require¬ 
ments  for  the  soldier  and  the  soldier's  role 
in  operating,  using,  or  maintaining  the 
materiel . 

(2)  Identification  of  human  factors  research 
required  to  support  the  training  requirement 
and  the  operational  concept. 

(3)  Identification  of  HFE  guidelines,  standards, 
processes,  or  criteria  and  other  documentation 
necessary  to  insure  that  operational  performance 
objectives  for  the  personnel-materiel  syst'-  .  can 
be  achieved  by  personnel  available  in  the 
organization  employing  the  system. 

(4)  Identification  of  training  devices  and  aids, 
and  special  training  requirements. 


The  Department  of  the  Navy  instruction  is  somewhat  dated, 
and  as  a  result,  terminology  does  not  always  match  current  use. 

A  term  formerly  associated  with  the  Demonstration  and  Validation 
Phase  is  contract  definition.  Once  again  they  simply  state  a 
requirement  for  this  activity. 
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As  a  minimum  this  will  involve  human  factors  inputs 
to  .  .  .  contract  definition  .... 


Similarly,  the  Department  of  the  Air  Force  regulation  calls 
for  activities  during  the  Demonstration  and  Validation  Phase. 

HFE  must  be  an  integral  part  of  .  .  .  exploratory 
(and)  advanced  .  .  .  development. 

Service  Requirements  for  Human 
Factors  R&D  m  the  Full-Scale 
Development  Phase 

The  intent  of  human  factors  activities  as  delineated  by 
service  requirements  in  the  Full-Scale  Development  Phase  is  to 
define  to  the  finest  degree  necessary  the  analyses  required  to 
support  detail  design  of  the  system.  This  phase  brings  together 
much  of  the  information  prepared  initially  in  earlier  phases  of 
system  development. 


The  Army  regulation  defines  specific  requirements  for 
information  to  support  system  development  in  this  phase.  Concern 
for  performance  of  the  human  in  the  system  is  manifested  by 
requirements  for  specific  data  (e.g.,  human  performance  relia¬ 
bility)  .  Human  factors  data  amenable  to  direct  usage  as  design 
recommendations  are  also  stressed. 

HFE  in  Full  Scale  Development  Phase.  Human  factors 
research  or  engineering  will  be  continued  to  insure 
the  timely  consideration  of  human  factors  in 
materiel  development  established  during  materiel 
concept  investigations,  advanced  development,  and 
the  preparation  of  a  Required  Operational  Capability 
(ROC) ,  Letter  of  Requirement  (LR) ,  Training  Device 
Requirement  (TDR) ,  or  Training  Device  Letter 
Requirement  (TDLR) .  Human  factors  research  or 
engineering  during  engineering  development  will 
include  more  detailed  task  analysis  and  further 
refinement  of  operator  and  maintenance  task 
sequences  as  the  materiel  design  is  finalized. 

These  task  sequences  will  be  used  to  determine 
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skill  requirements  and  training  implications 
and  their  impact  upon  organizational  structure. 
Human  engineering  characteristics  specified  in 
the  ROC,  LR,  TDR  or  TDLR  should  emphasize  system 
effectiveness,  human  performance  reliability, 
and  personnel  requirements.  Human  engineering 
standards  which  embody  well-tested  human  factors 
principles  or  prescribe  design  standardization 
to  minimize  cross-training  or  relearning  pi-oblems 
are  appropriate  as  guidelines  for  ROCs,  LRs, 

TDRs ,  and  TDLRs.  These  standards  should  not 
preclude  design  approaches  which  can  lead  to 
improved  performance  of  the  personnel-materiel 
system. 


The  Navy  instruction  simply  states  some  of  the  events  taking 
place  in  the  Full-Scale  Development  Phase. 

As  a  minimum  this  will  involve  human  factors  inputs 
»  to  .  .  .  engineering  development,  contractual 

statements  of  work,  engineering  change  proposals, 
and  test  and  evaluation  plans. 

Similarly,  the  Department  of  the  Air  Force  regulation  simply 
states  a  need  for  human  factors  activities  in  this  phase. 

HFE  must  be  an  integral  part  of  .  .  .  engineering 
development  projects  .... 

Service  Requirements  for  Human 
Factors  RSD  Through  Production 
and  Post-Production  Phases 

In  addition  to  the  previously  delineated  requirements  for 
>  human  factors  activities,  the  service  documentation  refers  to 

requirements  for  activities  throughout  the  balance  of  the  system 
life  cycle.  These  include  production  phases  (low  rate  initial 
production,  full  production,  and  deployment) ,  operation,  and 
»  retrofit.  As  post-production  deficiencies  are  identified,  human 

factors  activities  must  be  implemented  to  resolve  problems. 
Continuous  cognizance  of  personnel  and  training  requirements  must 
be  maintained  in  order  to  be  aware  of  and  resolve  problems  that 

• 
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may  arise  as  a  result  of  changes  or  other  factors.  The  intent 
is  to  convey  responsibility  for  human  factors  R&D  activities  to 
all  phases  of  system  development  and  operation. 

Informal  Documents 

Each  service  maintains  informal  documentation  which  involves 
substantial  methodology  applications,  as  stated  in  the  intro¬ 
duction  to  this  section.  Informal  documentation  consists  of  the 
various  handbooks,  guidebooks,  manuals,  etc.  that  the  services 
have  prepared  for  use  as  guidance  in  system  development  cycles. 
Whereas  the  formal  documents  {regulations  and  instructions) 
contain  detailed  requirements  and  responsibilities  for  human 
factors,  the  informal  documents  do  not.  Instead,  the  informal 
documents  provide  guidance  for  methodologies  (e.g.,  data  acqui¬ 
sition  and  analysis) ,  design  constraints  such  as  anthropometric 
data,  and  recommendations  for  what  may  be  found  in  a  good  human 
factors  program  plan.  The  materials  reviewed  tend  to  be  easily 
categorized  under  the  following  three  topics: 

1.  Human  Factors  in  System  Development 

2.  System  Design 

3.  Human  Factors  in  Test  and  Evaluation. 

Each  topic  will  be  briefly  discussed  in  subsequent  paragraphs. 

In  addition,  the  materials  applicable  to  each  will  be  briefly 
reviewed. 

Human  Factors  in  System  Development.  A  need  for  defining 
the  role  of  human  factors  in  military  system  development  has  been 
approached  by  the  services  in  detail  through  informal  documents. 
The  following  quote  states  the  need  most  succinctly: 

Man  is  a  system  component  and  as  such  deserves 
systematic,  specialized  attention.  System 
development  program  managers,  because  of  a  lack 
of  familiarity  with  or  interest  in  human  engi¬ 
neering,  have  sometimes  been  responsible  for 


relegating  human  engineering  to  a  ' nonessential ' 
status,  only  to  find  they  are  faced  with  costly 
redesign  at  a  later  date.  (Coburn,  1973) 

Coburn  proceeds  to  develop  a  complete  plan  for  human  factors 
in  the  Human  Engineering  Guide  to  Ship  System  Development .  With 
this  approach,  it  is  considered  imperative  that  human  factors  be 
integrated  into  a  program  early  in  system  development.  In  fact, 
this  is  a  prime  characteristic  of  all  system  development  oriented 
human  factors  guides  that  were  reviewed.  The  Naval  Air  Development 
Center  prepared  a  complete  management  approach  to  human  factors 
requirements  in  system  development  (Naval  Air  Development  Center, 
1974) .  This  document  encompasses  the  entire  system  acquisition 
cycle. 

Although  somewhat  dated,  a  guide  was  prepared  by  the  Army 
Human  Engineering  Laboratory  that  in  its  own  words  demonstrates 
a  concern  for  human  factors  integration  over  the  complete  system 
life  cycle.  "The  purpose  of  this  guide  is  to  explain  and  pro¬ 
mulgate  the  procedures  for  integrating  the  .  .  .  Human  Factors 
Engineering  Program  .  .  .  with  the  Life  Cycle  Management  Model 
for  Army  Systems"  (DA  Pamphlet  11-25).  (See:  Manpower  Resources 
Integration  Guide  for  Army  Materiel  Development,  Department  of 
the  Army,  1969.)  Since  this  document  related  to  the  predecessor 
of  the  present  LCSMM  pamphlet,  its  utility  may  well  be  limited. 
Nonetheless,  it  describes  a  complete  program  in  which  human 
factors  is  fully  a  part  of  system  development.* 

The  Air  Force  Systems  Command  has  prepared  a  design  handbook 
that  contains  expansions  of  requirements  for  human  factors 
throughout  the  system  development  cycle  (Air  Force  Systems  Command, 
1977).  Taking  a  more  generic  stance,  the  Human  Engineering  Guide 
to  Equipment  Design  (Van  Cott  &  Kinkade,  1972)  presents  the 


*It  is  understood  that  this  guide  is  presently  being  revised  and 
updated  by  the  Human  Engineering  Laboratory. 
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reasoning  behind  complete  integration  of  human  factors  throughout 
system  development.  All  of  the  above  articles  present  detailed 
methodologies  for  the  application  of  human  factors  to  the  devel¬ 
opment  of  systems.  Most  at  least  attempt  to  place  human  factors 
activities  within  the  structure  of  a  system  development  cycle. 

System  Design  Guidance.  Developing  a  data  base  from  which 
to  draw  human  factors  design  contributions  has  long  been  a  goal 
of  the  human  factors  specialist.  MIL-STD-1472  is  recognized  as  a 
primary  source  in  this  area.  There  are  other  documents  available 
which  have  expanded  upon  this  basic  resource  and  are  tailored 
with  additional  information  for  use  upon  specific  system  types. 

One  representative  document  in  this  regard  is  MIL-HDBK-759 
(Department  of  Defense,  1975).  Developed  for  use  by  the  Army, 
its  purpose  is  to  present  human  factors  data  which  are  sensitive 
to  unique  Army  materiel  requirements.  The  Air  Force  Systems 
Command  has  its  own  Design  Handbook  tailored  for  Air  Force  systems 
and  equipment  design  in  a  manner  similar  to  the  Army’s  (Air  Force 
Systems  Command,  1977).  Finally,  the  Human  Engineering  Guide  to 
Equipment  Design  presents  design  oriented  data  and  discussion. 

Human  Factors  in  Test  and  Evaluation.  Test  and  evaluation 
of  major  systems  has  been  a  large  concern  in  the  services. 
Extensive  documentation  of  human  factors  test  and  evaluation 
has  been  made.  The  Army  Human  Engineering  Laboratory  has  been 
responsible  for  numerous  test  and  evaluation  guides.  For  the 
most  part  these  have  been  superseded  by  the  guides  listed  next. 

The  most  important  guide  along  these  lines  contains  a  two  volume 
set  of  human  factors  test  procedures  and  guidance  for  test  data 
evaluation  (known  as  HEDGE — Human  Engineering  Data  Guide  for 
Evaluation)  (Perkins  et  al.,  1977).  The  test  procedure  volume 
includes  specific  test  procedures  and  sample  data  collection 
forms.  The  HEDGE  volume  covers  a  range  of  topics,  including: 
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test  conditions,  performance  tasks,  and  detailed  design  criteria, 
as  well  as  guidance  concerning  what  to  test.  Another  guide 
prepared  by  the  Human  Engineering  Laboratory  is  based  on  a  Data 
Item  Description  (DID)  entitled,  " Report (s)  of  Human  Factors 
Engineering  Test"  (Berson  &  Crooks,  1976) .  Guidelines  for 
conducting,  analyzing,  and  reporting  human  factors  tests  were 
developed  according  to  this  DID.  The  guide  also  provides  an 
application  procedure.  A  special  guide  for  evaluation  prepared 
by  the  Human  Engineering  Laboratory  was  tailored  to  Army  aviation 
requirements  for  human  factors  (Cassatt,  1965). 

The  Navy  has  developed  a  three  volume  manual  for  test  and 
evaluation  known  as  HFTEMAN  (Human  Factors  Test  and  Evaluation 
Manual)  (Department  of  the  Navy,  1976).  As  a  whole,  HFTEMAN 
covers  objectives,  methods,  procedures,  conditions,  analysis, 
and  reporting  techniques  of  test  and  evaluation.  In  addition, 
it  includes  topics  such  as  test  measurements,  criteria,  and 
standards  for  use  in  system  testing.  Finally,  the  Human 
Engineering  Guide  to  Equipment  Design  includes  a  discussion 
entitled,  "Human  Engineering  Test  and  Evaluation."  This  dis¬ 
cussion  extends  from  field  testing  through  laboratory  controlled 
experiments,  describing  inherent  advantages  and  disadvantages  of 
each. 

Conclusions 

The  purpose  of  the  preceding  discussion  of  service  require¬ 
ments  for  human  factors  R&D  in  military  system  acquisition  is  to 
support  the  view  that  policy  does,  in  fact,  exist  which  requires 
human  factors  activities  throughout  the  life  cycle  of  a  system. 
This  starts  with  mission  analysis  and  ends  when  the  system 
becomes  obsolete  and  is  withdrawn  from  use. 
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Three  points  can  be  drawn  to  conclude  this  discussion: 

1.  The  service  documentation  delineates  specific  require¬ 
ments  for  human  factors  activities. 

2.  Requirements  for  human  factors  activities  are  provided 
for  each  phase  of  the  military  system  acquisition 
process . 

3.  The  trend  in  formal  requirements  is  to  define  require¬ 
ments  and  responsibilities  for  human  factors  without 
placing  constraints  upon  the  methodology,  analysis,  and 
data  characteristics  used  in  research.  The  informal 
documentation,  on  the  other  hand,  covers  primarily  those 
topics  not  promulgated  in  the  formal  requirements  such 
as  methodological  options. 
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